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Abstract 

In this paper, we motivate the need for 
a publicly available, generic software 
framework for question-answering (QA) 
systems. We present an open-source QA 
framework QANUS which researchers can 
leverage on to build new QA systems eas¬ 
ily and rapidly. The framework imple¬ 
ments much of the code that will other¬ 
wise have been repeated across different 
QA systems. To demonstrate the utility 
and practicality of the framework, we fur¬ 
ther present a fully functioning factoid QA 
system Qa-Sys built on top of QANUS. 

1 Introduction 


of any derivative projects for everyone, QANUS is 
released under the Open Software License (OSL) 


v3.0. 


2 Related Work 


There has been previous efforts in generalis¬ 
ing the architecture of QA systems. Hirschman 
and Gaizauskas ( 2001| ) for example described a 
pipelined approach to QA (HG-01), where differ¬ 
ent stages are combined serially into a QA system. 
Figure [I] highlights the different stages in their 
pipeline vis-a-vis the stages found in QANUS. 
The informal correspondence between the various 
stages of the two pipelines are also shown in the 
figure. 


There has been much research into question¬ 
answering (QA) over the past decades. However 
the community is still lacking QA systems which 
are readily available for use. This translates into a 
high barrier of entry for researchers who are new 
to the field. The absence of easily accessible sys¬ 
tems also means that there is a lack of credible, 
reproducible baseline systems against which new 
QA systems can be evaluated. 

To address the highlighted limitations, we are 
releasing an open-source, Java-based, QA frame¬ 
work QANUS (pronounced KAY-NESS). QANUS 
is a framework on which new QA systems can 
be easily and rapidly developed. QANUS makes 
it easy to build new QA systems as only a min¬ 
imal set of components needs to be implemented 
on top of the provided framework. To demonstrate 
the utility and practicality of QANUS, a reference 
implementation of a QA system Qa-Sys has also 
been developed using the framework. Qa-Sys is 
also made available to the community. When it 
matures, it can serve as an accessible, reproducible 
baseline system for evaluations. 

To ensure the availability of the system to the 
community, as well as to maximise the benefits 
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Figure 1: Comparing pipeline stages of HG-01 
and Qanus. 


The architecture of HG-01 is slanted towards 
QA systems based on current state-of-the-art in¬ 
formation retrieval (IR) techniques. These tech¬ 
niques typically involve manipulating the lexical 
and syntactic form of natural language text and 
do not attempt to comprehend the semantics ex¬ 
pressed by the text. Systems which make use 


of these techniques (Hickl et al., 2007 Y. Chali, 


2007) have been able to perform ahead of their 


peers in the Text Retrieval Conference (TREC) 
QA tracks ( jDang et al., 2007j ). 



























In IR-based systems, answer processing re¬ 
volves around units of information stored in doc¬ 
uments. To reflect the importance of this organi¬ 
sation two separate stages (c) candidate document 
selection and (d) candidate document analysis are 
described in Hirschman’s architecture. Further, (f) 
answer generation is included as they considered 
interactive QA systems which could participate in 
a dialogue with end-users. 

Not all QA systems are IR-centric however, 
and interactive QA systems are likely not immi¬ 
nent given the limitations of natural language un¬ 
derstanding and generation. QANUS thus gen¬ 
eralises stages (c), (d) and (e) into one to avoid 
over-committing to any particular architecture or 
paradigm, and leaves out (f). 

Another important point of comparison is that 
QANUS is an implemented, functional QA archi¬ 
tecture whereas HG-01 serves mainly as a general 
discussion and introduction to the architecture of 
QA systems. 

Though few in numbers, some QA systems have 
previously been made available to the commu¬ 
nity. One such system is Aranea[[ (Lin, 2007). 
Aranea is a factoid QA system which seeks to 
exploit the redundancy of data on the web and 
has achieved credible performances at past TREC 
evaluations. Aranea is not designed however as 
a generic QA platform. We argue that a frame¬ 
work such as QANUS which is designed from the 
start with extensibility and flexibility in mind will 
greatly reduce the effort needed for any such cus¬ 
tomisation. 

Qanda by Mitr^] is another QA system 
which has featured in the TREC QA track. It has a 
project page on SourceForge. However currently 
only one module of the system is made available 
for download. We are at the time of writing unable 
to verify if there are plans for the release of the rest 
of the system in the near future. 

3 Qanus Framework 


(Lin, 2007 


programming code that will otherwise have been 
repeated across different QA systems. The frame¬ 
work can thus be likened to a foundation on top of 
which components can be added to obtain a com¬ 
plete QA system. 

Figure [2] illustrates a complete QA system built 
with the framework. The upper-half of the figure 
delineates clearly the key classes that constitute 
the four stages of the framework listed earlier. The 
bottom-half of the figure shows additional compo¬ 
nents that can be added to the framework to com¬ 
plete the QA system. For completeness, the input 
and output to the various stages of the system are 
also depicted as shaded boxes at the bottom of the 
figure. 

The top half of Figure [2] shows that each 
of the stages share a common architec¬ 
ture, composed of two main classes. The 
FrameworkController is responsible for 
directing the program flow and managing any 
input and output required or produced by the 
stage. It also invokes appropriate methods in 
the latter to process any input sent to the stage. 
The FrameworkEngine class provides the 
required processing that is needed on the various 
pieces of input to the stage. The processing that is 
required in each stage differs. For example, in the 
information source preparation stage, processing 
may involve part-of-speech tagging an input 
coipus, while in question processing, processing 
may instead be classifying the expected answer 
type of the posed questions. 

Due to space constraints, the individual inter¬ 
faces and function calls presented by QANUS are 
not explained in detail here. The full documenta¬ 
tion together with the source code for the frame¬ 
work are available at the QANUS download sit^J 

We briefly explain the operations that may be 
carried out in each stage. Note that this descrip¬ 
tion serves merely as a guide, and users of the 
framework have full flexibility in deciding the op¬ 
erations to be carried out at each stage. 


The QANUS framework adopts a pipelined ap¬ 
proach to QA. The pipeline consists of four stages 
executed serially.. The stages include (1) informa¬ 
tion source preparation, (2) question processing, 
(3) answer retrieval and (4) evaluation. Within 
the framework we have implemented much of the 

'Available for download at 

http://www.umiacs.umd.edu/~jimmylin/downloads/index.html 

z http://www.openchannelsoftware.org/projects/Qanda 


Information Source Preparation. In this 
stage, an information source from which answers 
are to be obtained is set up. The framework is 
not restricted to any particular type of informa¬ 
tion source. Depending on the required needs and 
specifications, the eventual information source can 
be as varied as a LUCENE0 index of the source 

3 http://junbin.com/qanus 

4 Open-source text search engine written in Java 
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Figure 2: Full QA system with QANUS framework and additional components. 


documents, a full-fledged ontology or the Inter¬ 
net. Any necessary pre-processing to set up the 
information source is done here. Note that this 
stage prepares static information sources. Using 
the Web dynamically as an information source is 
done in the subsequent answer retrieval stage. 

Question Processing. Typically, questions 
posed to the system need to be parsed and un¬ 
derstood before answers can be found. Neces¬ 
sary question processing is carried out here. Typ¬ 
ical operations here can include forming a query 
suitable for the information source from the posed 
questions, question classification to determine the 
expected answer type, as well as part-of-speech 
tagging and parsing. The outputs of these various 
operations are stored so that they can subsequently 
be used by the next stage in the QA pipeline. 

Answer Retrieval. The answer retrieval stage 
makes use of the annotations from the question 
processing stage, and looks up the information 
source for suitable answers to the posed questions. 
Incorporating candidate answers from dynamic 
sources, such as the Web or online databases, can 
also be incorporated here. Proper answer strings 
that can answer the questions are extracted in this 
stage. If desired, answer validation can be per¬ 
formed as well. 

Evaluation. With the three stages above, 
QANUS already provides the support necessary 
for a fully functional QA system. The evalua¬ 
tion stage is introduced to complement the ear¬ 
lier stages and ease the verification of the perfor¬ 
mance of the developed QA system. It is optional 
and may be omitted if desired. The evaluation 
stage cross-checks the answers computed previ¬ 


ously by the answer retrieval stage with a set of 
gold-standard answers. The results of the evalua¬ 
tion are then output for easy review. 

3.1 Additional Components 

The four stages of the QANUS framework es¬ 
tablish the flow of data through the entire QA 
pipeline, and form the backbone of any instanti¬ 
ated QA system. To realise the framework and ob¬ 
tain a fully functional QA system, additional com¬ 
ponents such as those shown in the bottom half of 
Figure [2] must be coupled to the QANUS frame¬ 
work. 

The classes in the framework enforce the re¬ 
quired interfaces that need to be adhered to by 
these additional components. By following the 
specified interfaces, any desired functionality can 
be plugged into the framework. 

To give a better picture of how these compo¬ 
nents can be easily added to the QANUS frame¬ 
work to complete a QA system, let us walk 
through an example for the question process¬ 
ing (QP) stage. From Figure [2j the mini¬ 
mum set of components that need to be im¬ 
plemented for QP include the QPController, 
QuestionlnputHandler, and QPEngine. 

QPController. QPController inherits from 
the QPFrameworkController component of 
the QANUS framework. This component is re¬ 
sponsible for initializing and integrating any text 
processing modules that will be used to process 
input questions with the framework. Suppose 
we want to perform part-of-speech tagging on 
the input questions, a part-of-speech component 
module needs to be created in QPController. 












































































QPController next notifies the QPEngine 

component about this part-of-speech tagger com¬ 
ponent. 

QuestionlnputHandler. This component is re¬ 
sponsible for reading in provided input questions. 
The implementation is thus dependent on how the 
input questions are formatted and presented. 

QPEngine. This component is derived from 
the QPFrameworkEngine component of the 
QANUS framework. It makes use of the ear¬ 
lier QuestionlnputHandler component to 
read in input questions, and invokes any text 
processing modules registered with it by the 
QPController to annotate the question text. 

It is useful to emphasise here the ease and flexi¬ 
bility provided by the QANUS framework: (1) The 
abstraction provided by the framework greatly re¬ 
duces the amount of code that needs to be written 
for a QA system. Only a minimal set of customi¬ 
sation needs to be carried out to complete the im¬ 
plementation of the QP stage. (2) The framework 
is sufficiently flexible to allow for a range of QA 
systems to be built. In the explanation here, only 
a part-of-speech tagger is described. Depending 
on requirements, other text processing algorithms 
and techniques can also be incorporated. 


4 Implementation of Qa-Sys 


To demonstrate the utility and practicality of the 
QANUS framework, we have developed a QA sys¬ 
tem, referenced to as Qa-Sys on top of the frame¬ 
work. The implementation of Qa-Sys is included 
when downloading QANUS to serve as an effec¬ 
tive reference implementation and help reduce the 
learning curve for researchers in using the frame¬ 
work. 

Qa-Sys is a fully functioning QA system de¬ 
veloped to run on the well-known dataset from the 
TREC 2007 QA track (Dang et al., 20071. Qa- 
Sys makes use of IR-based techniques to perform 
the QA task. As can be seen later, this includes 
making use of a text search engine to perform doc¬ 
ument lookup, as well as lexicon-based techniques 
including named entity recognition for answer re¬ 
trieval. An IR-based approach is adopted because 
it has been shown to turn in credible performances 
as explained earlier ([Hickl et al., 2007[ |Y. Chali, 
[2007> . 


Conforming to the description of the QANUS 
framework, Figure[3]shows the various classes that 
have been implemented as part of Qa-Sys. This 


figure is similar to Figure [2j which shows possi¬ 
ble components needed to obtain a complete QA 
system. 


Information Source Preparation. Similar to 
the participating machines of the TREC 2007 QA 
track, Qa-Sys makes use of the Aquaint-2 cor- 
pu^] which is stored in XME format. A XME 
parser AQUAINTXMLParser is written to inter¬ 
face the corpus with QANUS. LuceneWriter 
makes use of Fucene to build an index of the 
input coipus. We will subsequently make use of 
this index to retrieve documents relevant to posed 
questions in the later stages of the QA pipeline. 


Question Processing. In this stage, Qa- 
Sys attempts to classify the expected answer 
type of the input questions based on the tax¬ 
onomy described in Fi and Roth (2002) with 
QuestionClassif ier. We built the classifier 
used by training the Stanford Classifier (Manning 


and Klein, 20031 on the data described in Fi and 


Roth (2002). The classification assigned to each 
question is stored and passed on to the answer re¬ 
trieval stage. 


Answer Retrieval. To look up answers to 
the posed questions, Qa-Sys form a query out 
of the question by dropping stop-words found in 
the question. LuceneQuery uses this query to 
search through the Fucene index built earlier in 
the information source preparation stage. Doc¬ 
uments retrieved by the Fucene search engine 
are then broken down into individual passages. 
AnswerRetrieval scores each of these pas¬ 
sages using a variety of heuristics such as by tab¬ 
ulating the occurrences of the query terms within 
the passages. 

From the ranked passages, answer candidates 
are extracted depending on the expected answer 
type previously determined in question process¬ 
ing. For a question seeking a person name for 
example, a named entity recogniser fFinkel et al., 
) is used to extract candidate people names 
the ranked passages. For other expected an¬ 
swer types such as dates, hand-written regular ex¬ 
pressions are used to aid in the extraction of an¬ 
swer candidates. 

Finally, the answer candidates are ranked based 
again on a set of heuristics which include the prox¬ 
imity of the candidates within the ranked pas- 


2005 


from 


5 The corpus is not included with the download for Qa- 
Sys as it is the intellectual property of the LINGUISTIC Data 
Consortium. 
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Figure 3: Actual components implemented in Qa-Sys on top of the QANUS framework. 


sages to the query terms for example. The high¬ 
est ranked candidate is returned as the preferred 
answer. 

Evaluation. The evaluation stage provided by 
the QANUS framework makes it possible to eas¬ 
ily test the performance of Qa-Sys. Currently 
Qa-Sys supports only factoid questions, and so 
the evaluation metric used here is factoid accuracy 
( Dang et al., 2007) ), defined as: 

no. of correctly answer questions 
accuracy = -------—-.- 

total no. of test factoid questions 

which is implemented in 

FactoidAccuracyEvaluator. 

The top system in the TREC 2007 QA 
track LymbaPA 07 and the tenth-placed system 
QUANTA achieved accuracy scores of 0.706 and 
0.206 respectively. Qa-Sys currently obtains an 
accuracy of 0.119. 

There is room for improvement before Qa-Sys 
can catch up with the state-of-the-art. The current 
implementation is simplistic and does not do much 
processing of the input questions, nor does it per¬ 
form elaborate ranking of retrieved documents. As 
work on the system progresses and more sophis¬ 
ticated components are included into the system, 
Qa-Sys should be able to achieve better results. 

5 Future Work 

QANUS and Qa-Sys are currently under devel¬ 
opment. QANUS is relatively mature, having un¬ 
dergone several iterations of improvements and 
our work is now focused on improving the perfor¬ 
mance and functionalities of Qa-Sys. 


Performance. Conventionally, QA systems 
have been benchmarked against the systems par¬ 
ticipating in the TREC QA track. However re¬ 
cently the QA track has been dropped from both 
TREC and the Text Analysis Conference (TAC). 
As the years go by, the results from the QA track 
will age and become irrelevant. There is also a 
trend towards the use of the Web as an aid for QA. 
The Web is dynamic and any such QA system will 
likely not generate the same results in different in¬ 
stances of time. For useful benchmarking, it is 
thus important to be able to use a baseline sys¬ 
tem which makes use of the Internet at the same 
time instance as the QA system being compared 
to. Having access to such a baseline system is thus 
critical and essential. This is the niche that Qa- 
Sys serves to address.. When the performance 
of Qa-Sys catches up with the state-of-the-art, 
it will be a useful baseline system against which 
other QA systems can be evaluated against. 

To boost performance, more work needs to be 
done for the question processing and answer re¬ 
trieval stages. There are plans to include a query 
expansion component which will be helpful in 
boosting the precision of the documents retrieved 
by Lucene. To improve on answer retrieval, soft 
patterns as described in Cui et al. (2007) can re¬ 
place the current hard hand-written patterns used 
in the system. More advanced measures like the 
use of dependency relations (Cui et al., 2005) can 
also be adopted to improve on the current passage 
ranking implementation. 

List questions. Besides performance, it will 
also be useful to expand the functionalities of Qa- 
Sys. It does not handle list questions for the mo- 











































































ment. An implementation based on the use of re¬ 


dundancies found within the source text (Banko et 


al., 2002[ |Lin, 2001) is being considered. 

Internet front-end. An online demonstra¬ 
tion of Qa-Sys is currently hosted online]^] and 
supports querying over a pre-indexed Aquaint- 
2 corpus or the Internet. The answer retrieval 
component working with data from the Internet 
is rudimentary and lacks techniques to process 
the noise that accompanies data downloaded from 
the Internet. It will be useful to improve on 
this Internet-querying component by adding better 
post-processing over the retrieved data. 


6 Conclusion 


The lack of community-available QA systems has 
made it difficult to create new QA systems and 
perform comparisons across published studies. 
This motivated our work on an open-source QA 
framework QANUS. The framework implements 
much of the code needed for a QA system and re¬ 
duces the development effort needed to build new 
systems. It is carefully designed to be flexible and 
supports the use of a wide range of QA techniques. 

As a demonstration of the utility and practical¬ 
ity of QANUS, we have also implemented a fully 
functional factoid QA system Qa-Sys on top of 
the framework. Our goal is to improve Qa-Sys so 
that it will serve as a useful and accessible baseline 
to benchmark future QA systems and technologies 
against. Through this work, we hope to lower the 
high barriers of entry facing new QA researchers 
and reduce the time needed for them to begin pro¬ 
ductive research in this area. 
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